Image processing device and method, and program

ABSTRACT

Provided is an image processing device including an acquisition unit configured to acquire information on an imaging position and an imaging direction in units of frame images that constitute a moving image obtained through capturing by an imaging unit, a converted image generation unit configured to generate converted images having different imaging directions for each frame image that constitutes the moving image based on the frame image itself and preceding and succeeding frame images of the frame image, an evaluation value calculation unit configured to calculate an evaluation value for each converted moving image constituted by combining the converted image and the original frame image, the evaluation value being used to evaluate a blur between the converted images or between the original frame images, and a selection unit configured to select a converted moving image with less blur based on an evaluation value calculated by the evaluation value calculation unit.

CROSS-REFERENCE TO PRIOR APPLICATION

This application is a continuation of U.S. patent application Ser. No.14/652,546 (filed on Jun. 16, 2015), which is a National Stage PatentApplication of PCT International Patent Application No.PCT/JP2013/083296 (filed on Dec. 12, 2013) under 35 U.S.C. §371, whichclaims priority to Japanese Patent Application No. 2012-282611 (filed onDec. 26, 2012), which are all hereby incorporated by reference in theirentirety.

TECHNICAL FIELD

The present technology relates to image processing devices and methods,and programs, and more particularly, to an image processing device andmethod, and a program, capable of converting a blurred moving image intoa less blurred moving image.

BACKGROUND ART

In related art, solutions for a blurred image that occurs when a videocamera captures an image are classified into a mechanical approach ofcorrecting a blur caused by mechanical factors using a sensor such as agyro sensor when an image is captured and an electronic approach ofshifting an image using the optical flow techniques.

A technique for correcting a blurred image using combination of suchapproaches has been developed (see Patent Literature 1).

CITATION LIST Patent Literature

Patent Literature 1: JP 2011-217317A

SUMMARY OF INVENTION Technical Problem

However, the above-mentioned mechanical approach has a limit to thereduction in size, and thus, for example, it is difficult to beinstalled in a glasses-mounted video camera.

The above-mentioned electronic approach may eliminate the use ofmechanical components to achieve a more compact structure, but correctsimages one by one. When the images are continuously reproduced as amoving image, the correction is performed regardless of relationshipbetween the preceding and succeeding images. Thus, the moving imagereproduced as a whole is difficult to be viewed clearly, and the viewerexhibits symptoms similar to a so-called motion sickness.

The present technology is made in view of such circumstances, andespecially, when a captured moving image is reproduced, it is intendedto obtain a converted moving image with less blur by allowing imagingdirections to be a substantially constant direction.

Solution to Problem

According to an embodiment of the present technology, there is providedan image processing device including an acquisition unit configured toacquire information on an imaging position and an imaging direction inunits of frame images that constitute a moving image obtained throughcapturing by an imaging unit, a converted image generation unitconfigured to generate a plurality of converted images having differentimaging directions for each frame image that constitutes the movingimage based on the frame image itself and preceding and succeeding frameimages of the frame image, an evaluation value calculation unitconfigured to calculate an evaluation value for each converted movingimage constituted by combining the converted image and the originalframe image, the evaluation value being used to evaluate a blur betweenthe converted images or between the original frame images, and aselection unit configured to select a converted moving image with lessblur based on an evaluation value calculated by the evaluation valuecalculation unit.

A node difference calculation unit configured to calculate a differencebetween the converted image generated by the converted image generationunit and the original frame image as a node difference, and an edgedifference calculation unit configured to calculate a difference in atime series including the converted image generated by the convertedimage generation unit and the original frame image as an edge differencecan be further included. The evaluation value calculation unit cancalculate an evaluation value used to evaluate a blur for each ofconverted moving images based on each of a sum of the edge differenceand a sum of the node difference between the converted images or theoriginal frame images included in the converted moving image constitutedby combining the converted image and the original image.

The evaluation value calculation unit can calculate an evaluation valueused to evaluate a blur for each of the converted moving images byapplying a weight to each of the sum of the edge difference and the sumof the node difference between the converted images or the originalframe images included in the converted moving image constituted bycombination of the converted image and the original image and bycalculating a sum of the weighted values.

A weight input unit configured to input a weight to be applied to thesum of the edge difference and the sum of the node difference, a sum ofthe weight applied to each of the sum of the edge difference and the sumof the node difference being a value of 1 can be further included. Theevaluation value calculation unit can calculate an evaluation value usedto evaluate a blur for each of the converted moving images by applying aweight inputted by the weight input unit to each of the sum of the edgedifference and the sum of the node difference between the convertedimages or the original frame images included in the converted movingimage constituted by combination of the converted image and the originalimage and by calculating a sum of the weighted values.

The storage unit can also store information on an imaging directionbeing changed in accordance with the original frame image by having aplurality of predetermined angles of view and by rotating to a pluralityof angles in association with the converted image when the convertedimage is generated by the converted image generation unit. The imageprocessing device can further include a display image generation unitconfigured to generate a display image that displays a route on a mapbased on the converted moving image or the imaging position informationof the frame image of the moving image and displays by applying adifferent color to each imaging direction on the route based on theimaging direction information.

A specifying unit configured to specify a position on the routedisplayed on the map on the display image, and a reproduction unitconfigured to reproduce the converted moving image that constitutes theroute or the moving image from a frame image including information on animaging position corresponding to a position specified by the specifyingunit can be further included.

The converted image generation unit can generate a converted image byconverting each frame image that constitutes a moving image stored inthe storage unit into a plurality of images having the different imagingdirections by having a plurality of predetermined angles of view and byrotating to a plurality of angles by a technique including an angle ofview shift, homography transformation, and viewpoint composition, usingthe frame image itself and preceding and succeeding frame images of theframe image.

The converted image generation unit can generate a plurality ofhigher-resolution converted images having the different imagingdirections for each frame image that constitutes a moving image storedin the storage unit by using a pixel generated by a phase shift equal toor smaller than a pixel based on the frame image itself and precedingand succeeding frame images of the frame image.

The imaging unit configured to capture the moving image can be furtherincluded. The acquisition unit can store the information on the imagingposition and the imaging direction of the frame image in associationwith each of the frame images.

According to an embodiment of the present technology, there is providedan image processing method including acquiring information on an imagingposition and an imaging direction in units of frame images thatconstitute a moving image obtained through capturing by an imaging unit,generating a plurality of converted images having different imagingdirections for each frame image that constitutes the moving image basedon the frame image itself and preceding and succeeding frame images ofthe frame image, calculating an evaluation value for each convertedmoving image constituted by combining the converted image and theoriginal frame image, the evaluation value being used to evaluate a blurbetween the converted images or between the original frame images, andselecting a converted moving image with less blur based on a calculatedevaluation value.

According to an embodiment of the present technology, there is provideda program for causing a computer to execute processing including anacquisition step of acquiring information on an imaging position and animaging direction in units of frame images that constitute a movingimage obtained through capturing by an imaging unit, a converted imagegeneration step of generating a plurality of converted images havingdifferent imaging directions for each frame image that constitutes themoving image based on the frame image itself and preceding andsucceeding frame images of the frame image, an evaluation valuecalculation step of calculating an evaluation value for each convertedmoving image constituted by combining the converted image and theoriginal frame image, the evaluation value being used to evaluate a blurbetween the converted images or between the original frame images, and aselection step of selecting a converted moving image with less blurbased on an evaluation value calculated by processing of the evaluationvalue calculation step.

In an embodiment of the present technology, information on an imagingposition and an imaging direction in units of frame images thatconstitute a moving image obtained through capturing by an imaging unitis acquired, a plurality of converted images having different imagingdirections for each frame image that constitutes the moving image basedon the frame image itself and preceding and succeeding frame images ofthe frame image are generated, an evaluation value being used toevaluate a blur between the converted images or between the originalframe images for each converted moving image constituted by combiningthe converted image and the original frame image, and a converted movingimage with less blur based on a calculated evaluation value is selected.

The image processing device according to an embodiment of the presenttechnology may be a stand-alone device or may be a block that performsinformation processing.

Advantageous Effects of Invention

According to the embodiments of the present technology, a blurred movingimage can be converted into a less blurred moving image by an easyoperation.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an exemplary general configuration ofan embodiment of an image processing system to which the presenttechnology is applied.

FIG. 2 is a diagram illustrating an exemplary appearance configurationof a glasses-mounted camera shown in FIG. 1.

FIG. 3 is a diagram illustrated to describe a functional configurationfor implementing the glasses-mounted camera shown in FIG. 1.

FIG. 4 is a diagram illustrated to describe a functional configurationfor implementing a reproduction terminal shown in FIG. 1.

FIG. 5 is a diagram illustrated to describe a functional configurationfor implementing a server shown in FIG. 1.

FIG. 6 is a flowchart illustrated to describe an imaging and recordingprocess.

FIG. 7 is a diagram illustrated to describe a configuration of acaptured moving image file.

FIG. 8 is a flowchart illustrated to describe a reproduction imageselection image displaying process.

FIG. 9 is a diagram illustrated to describe an exemplary display of MyMap that is a reproduction image selection image.

FIG. 10 is a diagram illustrated to describe how to operate a correctionknob.

FIG. 11 is a flowchart illustrated to describe a route correctionprocess.

FIG. 12 is a diagram illustrated to describe a node difference that is adifference between an original image and a converted image.

FIG. 13 is a diagram illustrated to describe an edge difference that isa difference between the preceding and succeeding images in time seriesof original images or converted images.

FIG. 14 is a diagram illustrated to describe a conversion processbetween frame images of a moving image file, which can be processed as agraph optimization problem.

FIG. 15 is a diagram illustrated to describe an example of cutting offthe region where an image is collapsed due to a converted image.

FIG. 16 is a flowchart illustrated to describe a reproduction process.

FIG. 17 is a diagram illustrated to describe an exemplary configurationof a general-purpose personal computer.

DESCRIPTION OF EMBODIMENTS

<Exemplary Configuration of Imaging System>

FIG. 1 is a diagram illustrating an exemplary configuration of an imageprocessing system to which the present technology is applied. The imageprocessing system 1 shown in FIG. 1 is configured to includeglasses-mounted cameras 11-1 to 11-N, reproduction terminals 12-1 to12-M, a network 13, and a server 14. The glasses-mounted cameras 11-1 to11-N are simply referred to as a glasses-mounted camera 11, and thereproduction terminals 12-1 to 12-M are simply referred to as areproduction terminal 12, as long as it is not especially necessary todistinguish between the components with the same name. This is similarlyapplicable to other components.

In the image processing system 1 shown in FIG. 1, a moving imagecaptured by the glasses-mounted camera 11 is transmitted to the server14 and is stored therein. When a user of the glasses-mounted camera 11operates the reproduction terminal 12 and accesses the server 14, thestored moving image is reproduced and is viewed by the user.

More specifically, the glasses-mounted camera 11 captures a movingimage, adds imaging position and imaging direction information to eachof frame images that constitute the captured moving image, and generatesa moving image file having a time length of approximately 1 minute. Theglasses-mounted camera 11 then transmits the resulting moving image fileto the server 14 via the network 13. The server 14 assigns an index tothe moving image file to which the imaging position and imagingdirection in frame units, which is supplied from the glasses-mountedcamera 11, are added, and then the server 14 stores the indexed movingimage file. The server 14, when accessed by the reproduction terminal 12operated by the user, generates a corrected moving image with less blurbased on the imaging direction information and generates a display imagethat displays the user's moving path on a map, with the user wearing theglasses-mounted camera 11 and capturing a moving image, based on theimaging position information. The server 14 distributes the generatedimages together to the reproduction terminal 12. The corrected movingimage generated by the server 14 is obtained by correcting the storedimage into a less blurred image. When the reproduction terminal 12obtains a map image on which a route corresponding to the moving path ismarked, the reproduction terminal 12 displays the map image. When anyone position in the route corresponding to the moving path on the mapimage is selected, the reproduction terminal 12 reproduces and displaysa moving image starting from the image captured at the imaging positioncorresponding to the selected position. Thus, the user of theglasses-mounted camera 11 can specify any moving path on the map withoutconsidering information such as imaging capabilities or image-capturedtime. This allows the user to enjoy viewing images captured by the userwhile searching an image desired to be viewed.

<Exemplary Appearance Configuration of Glasses-mounted Camera>

An exemplary appearance configuration of the glasses-mounted camera 11will be described with reference to FIG. 2. The glasses-mounted camera11 may be such a thing as shown in FIG. 2, and when the user wears itlike wearing eyeglasses, an imaging unit 32 captures a view imageobserved by the user through a pair of lenses 31-1 and 31-2.

The glasses-mounted camera 11 is configured to include the lenses 31-1and 31-2, the imaging unit 32, and a control unit 33. The lenses 31-1and 31-2 may be an ordinary spectacle lens. The imaging unit 32 may becomposed of a charge-coupled device (CCD) or complementary metal oxidesemiconductor (CMOS) sensor. The imaging unit 32 compresses a capturedmoving image according to a predetermined format and supplies thecompressed moving image to the control unit 33. The control unit 33generates a moving image file having a time length of approximately 1minute from the moving images captured sequentially using variouscomponents shown in FIG. 3, and transmits the generated moving imagefile to the server 14.

<Exemplary Configuration of Implementing Glasses-Mounted CameraFunctions>

An exemplary configuration for implementing functions of theglasses-mounted camera 11 will be described with reference to FIG. 3.The glasses-mounted camera 11 is provided with the imaging unit 32 andthe control unit 33, as described with reference to FIG. 2. The controlunit 33 is configured to include a recording unit 51, a globalpositioning system (GPS) 52, a real time clock (RTC) 53, an earth's axissensor 54, a gyro sensor 55, a controller 56, and a communication unit57. The recording unit 51 temporarily records the moving images suppliedsequentially from the imaging unit 32. In this case, the recording unit51 acquires imaging direction information that is obtained from positioninformation supplied from the GPS 52, time information supplied from theRTC 53, direction information of the earth's axis supplied from theearth's axis sensor 54, information on yaw, pitch, and roll angularvelocity supplied from the gyro sensor 55. The recording unit 51 recordsthe acquired information in association with each frame image thatconstitutes the captured moving image. More specifically, the recordingunit 51 estimates an imaging direction of the image unit 32 based on theinformation on yaw, pitch, and roll angular velocity and the directioninformation of the earth's axis detected by the earth's axis sensor 54.The recording unit 51 stores the estimated imaging direction informationin association with the frame image. The recording unit 51 generates amoving image file at predetermined time intervals from the moving imagessupplied sequentially from the imaging unit 32, and supplies thegenerated moving image file to the controller 56.

The GPS 52 receives radio waves from a satellite (not shown) andcalculates latitude and longitude information to be used as positioninformation on the earth. The GPS 52 supplies the calculated informationto the recording unit 51 as information indicating an imaging position.The RTC 53 manages an actual time and supplies information indicatingthe image-captured time to the recording unit 51. The earth's axissensor 54 detects an inclination relative to the earth's axis in thedirection of the glasses-mounted camera 11 and supplies the detectedresults to the recording unit 51. The gyro sensor 55 detects yaw, pitch,and roll angular velocity and supplies the detected results to therecording unit 51.

The controller 56 controls the overall operation of the control unit 33of the glasses-mounted camera 11. The controller 56 controls therecording unit 51 to generate a moving image file having a predeterminedtime length, for example, a time length of approximately 1 minute basedon moving image data supplied from the imaging unit 32, and acquires thegenerated moving image file. The controller 56 controls thecommunication unit 57 to transmit the generated moving image file to theserver 14 via the network 13.

<Exemplary Configuration for Implementing Reproduction TerminalFunctions>

An exemplary configuration for implementing functions of thereproduction terminal 12 will be described with reference to FIG. 4.

The reproduction terminal 12 is configured to include a control unit 71,a display unit 72, an operation unit 73, a communication unit 74, and adisplay image storage unit 75. The control unit 71 controls the overalloperation of the reproduction terminal 12. The display unit 72 mayinclude a liquid-crystal display (LCD), an organic electro-luminescence(EL) display, or other display devices. The display unit 72 displays adisplay image composed of a map image supplied from the server 14 ordisplays a reproduction image composed of a moving image file suppliedfrom the server 14. The operation unit 73 may include a keyboard, amouse, or operation buttons. When the user operates the operation unit73 to input various commands and instructions, the operation unit 73supplies an operation signal corresponding to the user's operation tothe control unit 71. The communication unit 74 transmits and receivesdata and commands to and from the server 14 via the network 13 under thecontrol of the control unit 71. The display image storage unit 75 storesa display image used as user interface (UI) that is necessary to allow amoving image file stored in the server 14 to be selected and reproduced.An example of the display image to be stored includes a map image thatis necessary as an interface image.

<Exemplary Configuration for implementing Server Functions>

An exemplary configuration for implementing functions of the server 14will be described with reference to FIG. 5.

The server 14 is configured to include a communication unit 91, arecording unit 92, a storage unit 93, and a reproduction management unit94. The communication unit 91 transmits and receives data, a movingimage file, and various commands from and to the glasses-mounted camera11 and the reproduction terminal 12 via the network 13. The recordingunit 92 is provided with an indexing unit 101. The recording unit 92sequentially receives a moving image file having a time length ofapproximately 1 minute transmitted from the glasses-mounted camera 11.The recording unit 92 allows the indexing unit 101 to generate an indexfor each moving image file based on a frame image included in the movingimage file and information about the imaging position and imagingdirection added to the frame image, and to assigns a generated index toeach moving image file. The recording unit 92 then stores an indexedmoving image file in the storage unit 93.

The reproduction management unit 94 generates a display image thatdisplays a route indicating the moving path of the user based on theimaging position of the captured image on a map stored in advance whenthe reproduction management unit 94 is instructed by the reproductionterminal 12 to reproduce the captured moving image file via thecommunication unit 91. The generation of the display image is performedbased on the imaging position and imaging direction stored in frameimage units of the moving image file stored in the storage unit 93. Thereproduction management unit 94 corrects an image of the moving imagefile stored in the storage unit 93 to be an image with reduced blur andsupplies the corrected moving image file to the reproduction terminal12.

More specifically, the reproduction management unit 94 is configured toinclude a My Map creation unit 111, an imaging direction coloring unit112, an object recognition unit 113, a map data storage unit 114, aroute editing unit 115, a converted moving image file generation unit116, a node difference calculation unit 117, an edge differencecalculation unit 118, an evaluation value calculation unit 119, and aconverted moving image file selection unit 120.

The My Map creation unit 111, when instructed by the reproductionterminal 12 to reproduce a moving image file via the communication unit91, reads the moving image file instructed to be reproduced from thestorage unit 93. The My Map creation unit 111 specifies a routeindicating the moving path of the user and generates a display image,called My Map, to be displayed on a map stored in the map data storageunit 114 while capturing an image that constitutes the moving imagefile. The specification and generation by the My Map creation unit Illare performed based on information indicating the imaging positionstored for each frame image that constitutes the moving image fileinstructed to be reproduced. The My Map is distributed to thereproduction terminal 12 together with the moving image file and isdisplayed on the reproduction terminal 12. The displayed map is dividedinto sections with a step size according to the number of frame imagescaptured at the same position. When a position on the map is pointed byusing a pointer or the like, a frame image corresponding to the pointedposition is displayed or a moving image file from the frame imagecaptured at the corresponding imaging position is reproduced.

The imaging direction coloring unit 112 applies different colors to thecorresponding positions on a route displayed on My Map used as thedisplay image described above, based on the imaging directioninformation stored in association with frame images included in themoving image file. This process allows a frequent change in colors on aroute to indicate a frequency change in imaging directions. Thus, whenthe moving image file is reproduced and is viewed by the user withoutany modification, blur will be more visible and the user who views themoving image file feels motion sickness. On the other hand, when colorson a route remain unchanged in some sections, it means that the imagingdirection is kept constant in the sections. Thus, when the moving imagefile is reproduced and is viewed by the user without any modification,blur will be less visible and the user who views the moving image filedoes not feel motion sickness.

The object recognition unit 113 recognizes an object in an image foreach frame image, extracts the recognized object image, adds informationon the recognized object to the object image, and stores it in thestorage unit 93 in a classified fashion. The object images constitutethe XrossMediaBar, which will be described later, and are displayed in aselectable state for each object when the display image of My Mapdescribed above is generated.

The map data storage unit 114 stores data of a map image on the earthand supplies the map data to the My Map creation unit 111 when the MyMap creation unit 111 creates a My Map.

The route editing unit 115 edits a route obtained by plottinginformation that indicates an imaging position on the My Map based onthe imaging position information included in the header for each frameimage included in the moving image file stored in the storage unit 93.

The converted moving image file generation unit 116 generates multiplekinds of converted images by performing an angle of view conversion orrotation processing for converting the imaging direction with the use ofthe preceding and succeeding frame images or the like for each frameimage included in the moving image file stored in the storage unit 93.The converted moving image file generation unit 116 then generates theconverted moving image file composed of converted images and stores theconverted moving image file in the storage unit 93. In this time, theconverted moving image file generation unit 116 stores the convertedmoving image file by adding information on the imaging direction changedby the conversion processing to the converted frame image.

The node difference calculation unit 117 calculates a node difference byobtaining a difference between the converted image and an original imagebefore conversion for each of the converted images. The node differencecalculation unit 117 calculates a sum of difference between nodes inconverted images or original images of all patterns included in theconverted moving image file, which is generated by combining convertedimages or original images in various patterns in time series.

The edge difference calculation unit 118 calculates an edge differenceby obtaining a difference between converted images or original images,which are arranged in time series. The edge difference calculation unit118 calculates a sum of edge difference between converted images ororiginal images of all patterns included in the converted moving imagefile, which is generated by combining converted images or originalimages in various patterns in time series.

The evaluation value calculation unit 119 obtains an evaluation value ofeach of the converted moving image file by calculating a sum total ofthe sum of difference between nodes and the sum of difference betweenedges, in the converted moving image file of the pattern which isgenerated by combining converted images or original images in variouspatterns in time series. In this case, the evaluation value calculationunit 119 acquires weight information supplied from the reproductionterminal 12 and applies a weight to each of the node difference sum andthe edge difference sum to calculate an evaluation value.

The converted moving image file selection unit 120 selects a movingimage file having the smallest evaluation value from among all of theconverted moving image files of the pattern that is generated bycombining converted images or original images in various patterns intime series. The converted moving image file selection unit 120transmits the selected moving image file to the reproduction terminal12, which requests to reproduce the moving image file, via thecommunication unit 91.

A prohibition editing unit 121 searches a region to be prohibited frombeing reproduced in the images instructed to be reproduced for eachframe image that constitutes the moving image file instructed to bereproduced, based on the imaging position and time information added tothe frame image. The prohibition editing unit 121 performs a process forchanging the searched region into, for example, a mosaic image. In otherwords, in the image captured by the glasses-mounted camera 11, aprohibited region may include a region containing information such aspersonally identifiable information, a region in which military secretor the like specified based on position information is seemed to becaptured, and a region that may have a problem if disclosure is made ata predetermined time.

<Imaging and Recording Process>

The imaging and recording process in the image processing system shownin FIG. 1 will be described with reference to FIG. 6. This processstarts when an operation unit (not shown) of the glasses-mounted camera11 is operated and the imaging unit 32 is instructed to start capturingan image.

In step S11, the imaging unit 32 captures an image and supplies thecaptured image data to the recording unit 51.

In step S12, the GPS 52 receives radio waves from a satellite (notshown), calculates latitude and longitude to be used as a position onthe earth, and supplies the position information on the earth includingthe latitude and longitude obtained as the calculation results to therecording unit 51.

In step S13, the earth's axis sensor 54 measures an inclination relativeto the direction of the earth's axis and supplies information on theinclined angle of the glasses-mounted camera 11 relative to thedirection of the earth's axis obtained as the measurement results to therecording unit 51.

In step S14, the gyro sensor 55 detects yaw, pitch, and roll angularvelocity in the glasses-mounted camera 11 and supplies information oneach angular velocity obtained as the detected results to the recordingunit 51. This process allows the recording unit 51 to calculate andstore an imaging direction based on the inclined angle relative to theearth's axis and the yaw, pitch, and roll angular velocity.

In step S15, the RTC 53 reads current time information and supplies theread current time information to the recording unit 51.

In step S16, the recording unit 51 acquires information on theimmediately preceding moving image file.

In step S17, the recording unit 51 records moving image data in frameimage units. In this case, the recording unit 51 sets the imagingposition indicating the position information on the earth, theimage-captured time indicating the supplied time, the imaging direction,and the immediately preceding moving image file information as a headerof each frame at the timing of capturing each frame image, and storesthese information in the moving image file to be accumulated in asequential manner. The recording unit 51 records an address on thenetwork being in use, a media access control (MAC) address or the likeas information for identifying itself on the header of each frame image.

In step S18, the recording unit 51 determines whether a predeterminedtime is elapsed based on a difference between the current time suppliedfrom the RTC 53 and the time at which a new moving image file starts tobe stored. In step S18, if a predetermined time is determined to beelapsed, the process proceeds to step S19.

In step S19, the recording unit 51 determines that a predetermined timeis elapsed, and then the recording unit 51 closes the stored movingimage file and supplies the generated moving image file to thecontroller 56. The controller 56, when receiving the supplied movingimage file, controls the communication unit 57 to supply the receivedmoving image file to the server 14 via the network 13. In step S18, ifit is determined that a predetermined time is not elapsed, the processin step S19 is skipped.

In step S20, the imaging unit 32 determines whether an instruction toterminate the imaging operation of a moving image is issued based on anoperation signal from an operation unit (not shown). If it is determinedthat termination is instructed, the process ends. On the other hand, instep S20, if it is determined that termination is not instructed, theprocess returns to step S11 and the subsequent steps are repeated untilthe instruction of termination is issued. In step S18, if it isdetermined that a predetermined time is not elapsed, the process in stepS19 is skipped.

The moving image files are recorded at a predetermined time interval Tof approximately one minute, and the moving image files are configuredto overlap with each other by only a predetermined time t, as shown inFIG. 7. In other words, in FIG. 7, a moving image file V1 is configuredas a moving image file captured by only the length of the predeterminedtime T. The next moving image file V2 is configured so that the time tof, for example approximately 10 seconds, before the moving image fileV1 ends has the same moving image data as the time t after the movingimage file V2 starts. The subsequent moving image file V3 is configuredso that the time t before the moving image file V2 ends has the samemoving image data as the time t after the moving image file V3 starts.This configuration of moving image files allows only a portion with atime length of 40 seconds in the moving image file to be lost even whena moving image file, which is being transmitted to the server 14, doesnot reach the server 14 due to any accident on the network 13. It isalso possible to reduce the load on the network 13 caused by excessivelylarge capacity of the moving image file.

On the other hand, in the server 14, in step S41, the recording unit 92controls the communication unit 91 and determines whether a moving imagefile is transmitted from any one glasses-mounted camera 11 on thenetwork 13. The recording unit 92 then repeats this process until themoving image file is transmitted. In step S41, for example, if themoving image file is transmitted in the process of step S19, the processproceeds to step S42.

In step S42, the recording unit 92 controls the communication unit 91 toacquire the transmitted moving image file. The recording unit 92controls the indexing unit 101 to read information of frame imagesincluded in the moving image file and to read information foridentifying a glasses-mounted camera 11 and information including animage, an imaging position, and an image-captured time, which arenecessary for indexing. The frame image to be read for indexing includesa typical person's face image, a typical landscape, a typical building,or the like. The imaging position and image-captured time to be used forindexing may be, respectively, the imaging position and animage-captured time of the leading frame image.

In step S43, the indexing unit 101 generates an index based on theinformation read as index information.

In step S44, the indexing unit 101 adds the generated index informationto the transmitted moving image file. The recording unit 92 stores theindexed moving image file in the storage unit 93.

This process allows the image captured by the glasses-mounted camera 11to be recorded on the server 14 as an indexed moving image file having atime length of approximately one minute.

<Reproduction Image Selection Image Displaying Process>

A reproduction image selection image displaying process will bedescribed with reference to the flowchart shown in FIG. 8. This processis based on the assumption that the moving image file is stored in thestorage unit 93 of the server 14 by the imaging and recording processdescribed with reference to the flowchart shown in FIG. 6.

In step S61, the control unit 71 reads a map image stored in the displayimage storage unit 75 and display surrounding images of the homeposition set in advance.

In step S62, the control unit 71 reads specified position information ona map. In the initial process, information on the home position is readas the specified position information. In the subsequent processes, aposition specified by the user who operates the reproduction terminal 12is read as the specified position information by the operation of anoperation unit (not shown).

In step S63, the control unit 71 determines whether or not the specifiedposition information is the initial position information or newlyspecified position information. For example, it is determined that thespecified position information is the initial position information ornewly specified position information, the process proceeds to step S64.

In step S64, the control unit 71 controls the communication unit 74 totransmit the specified position information to the server via thenetwork 13.

In step S81, the reproduction management unit 94 controls thecommunication unit 91 to determine whether the specified positioninformation is transmitted. In step SS1, for example, if it isdetermined that the specified position information is transmitted in theprocess of step S64, the process proceeds to step S82.

In step S82, the reproduction management unit 94 controls the My Mapcreation unit 111 to read map data near the specified positioninformation from among the map data stored in the map data storage unit114 based on the specified position information, and generates areference image for the My Map. The reference image for the My Map maybe a map image display field m shown in FIG. 9. In the map image displayfield m shown in FIG. 9, a street named “A avenue” running from top tobottom of the map is displayed on the left side in the figure. In themap, a crossroad is displayed in a road going from the branch near themiddle of the street to the right side in the figure, and a buildingnamed “C building” is displayed in a corner of the crossroad. Astructure named “D house” is displayed in the further right side in thefigure along the road beyond the crossroad with the “C building” at thecorner. By turning right from a T-junction with the “D house” at thecorner and then going to the left side, a highway named “B highway” isdisplayed on the map.

In step S83, the My Map creation unit 111 sets the latitude andlongitude of the specified position as a target position to beprocessed.

In step S84, the My Map creation unit 111 searches index information ofthe moving image file stored in the storage unit 93, and measures thenumber f of moving image files at the surrounding positions, based onthe latitude and longitude information in target position information tobe processed.

In step S85, the My Map creation unit 111 sets a step width F of thenumber of pixels proportional to 1/f for a given range near the targetposition to be processed.

In step S86, the My Map creation unit 111 divides a given range near thetarget area to be processed by the step width F.

In step S87, the My Map creation unit 111 determines whether anunprocessed position is in the selected map image. If an unprocessedposition is determined to be in the map image, the process proceeds tostep S88.

In step S88, the My Map creation unit 111 sets any of the unprocessedpositions as a target position to be processed, and then the processreturns to step S84. In other words, the process in steps S84 to S88 isrepeated until all the positions in the image read as a map image areset as the target position to be processed and the image is divided bythe step width F according to the number of moving image files. When allof the positions are set as the target position to be processed, none ofthe unprocessed positions is assumed to be within the map image in stepS87, and then the process proceeds to step S89.

In other words, as shown in FIG. 9, on the map, a region Z1 near aposition Y and a region Z2 near the position P or X are divided intosmaller regions by a smaller step width as indicated by the dotted line.This division is performed according to the number of moving image filesregistered in association with the positions on the map image. Thisindicates that numerous moving image files are recorded. On the otherhand, in the regions Z3 and Z4, and regions other than the regions Z1and Z2, there is no area bounded substantially by the dotted line. Thisindicates that a moving image file does not substantially exist. Inother words, this display makes it easy to view a position at whichnumerous moving image files exist and a position at which a moving imagefile does not exist.

As shown in FIG. 9, when the reproduction image selection image calledMy Map is displayed on the display unit 72 of the reproduction terminal12 and any one position in a mass of regions surrounded by the dottedline is specified using a pointer or other device (not shown), aninstruction to reproduce a moving image file corresponding to thespecified position is supplied to the server. The server 14 sequentiallytransmits frame images included in the corresponding moving image filein response to the reproduction instruction. The reproduction terminalthen reproduces the moving image file. In this case, at the time ofselection before the reproduction is instructed, information including astill image or image-captured time that constitutes a header may bedisplayed.

In step S89, the route editing unit 115 plots information indicating animage-captured position of a frame image in which the image-capturedposition exists on a map image displayed on the My Map in associationwith time information. The route editing unit 115 creates a route byobtaining a moving path of the user wearing the glasses-mounted camera11 that captures the frame image included in the moving image file anddisplays the route on the display screen. In this time, the imagingdirection coloring unit 112 reads information indicating the imagingdirection included in the header of each of the plotted frame imagesthat constitute the route created by the route editing unit 115 and addsdifferent colors depending on the imaging direction to the route.

In other words, for example, this process allows a route L to bedisplayed, as shown at positions X to Y. As shown in FIG. 9, the route Lis displayed by adding different colors to each imaging direction of theframe images included in the moving image file at each position. As aresult, the range in which the color on the route L is greatly changedindicates that the imaging direction is frequently changed. Thus, ingeneral, the imaging direction frequently changed at the time of viewingcauses the moving image to induce motion sickness. On the other hand,the range with a small change in colors indicates that the change in theimaging directions has low frequency. Thus, in general, the imagingdirection infrequently changed at the time of viewing causes the movingimage easy to be viewed and difficult to induce motion sickness.

The change in the imaging directions may be corrected by moving acorrection knob M shown in the upper right of FIG. 9 to the left andright on a correction scale shown by a horizontal arrow. In other words,as shown in FIG. 10, as the correction knob M is moved to the left inthe figure, the route becomes an image similar to the frame imageincluded in the inputted moving image file as shown by the state of aroute J1. Thus, it is reproduced without any modification even when theimaging direction is frequently changed. In this case, the frame imagesimilar to the state obtained upon capturing may be used and thusindividual frame images are difficult to be collapsed, but when thechange in the imaging direction is large, the moving image is difficultto view and induces motion sickness.

On the other hand, as shown in FIG. 10, as the correction knob M ismoved to the right in the figure, the route becomes an image dissimilarfrom the frame image included in the inputted moving image file as shownby the state of a route 13. In this case, the correction of the imagingdirection makes the individual frame images to be a state in which it iseasy to be collapsed, but the image imaging directions are aligned tosome extent and thus it is relatively easy to view the moving image. Inthe state of routes J2 and J3 shown in FIG. 10, some portions of theroute may be cut out because the collapsed image is intended to beremoved. These routes J1 to J3 indicate that actual image-capturedpositions, that is, the route corresponding to the moving path of theuser wearing the glasses-mounted camera 11 is close to a straight lineby unifying the imaging directions

A route correction process performed by operating the correction knob Mwill be described later in detail with reference to the flowchart ofFIG. 11.

In step S90, the object recognition unit 113 classifies and extracts thetype of objects included in the frame image that constitutes the movingimage file at the specified position.

In step S91, the object recognition unit 113 extracts time informationfrom the object image extracted from each of the objects classified fromthe frame image included in the moving image file at the specifiedposition and classifies the extracted time information by every time.

In step S92, the My Map creation unit 111 associates the object imagesclassified by the object type of the image by the object recognitionunit 113 with the time information, and the My Map creation unit 111displays the associated object images using the XrossMediaBar as shownin FIG. 9. The XrossMediaBar is constituted by images P11 to P16 andimages P21 to P26, shown in FIG. 9. In other words, the images P11 toP16 are set in the vertical direction on the basis of time. In FIG. 9,the images P11 to P16 are the same object of house and are classified bytime from 12 AM (12:00) to 5 PM (17:00). It is possible to display theimages recognized as the same object in time series by moving it in thevertical direction.

The images P11 and P21 to P26 are displayed in the horizontal directionfor each type of objects classified according to a predetermined rule.It is possible to select the classified objects by moving it in thehorizontal direction. Thus, in FIG. 9, a house image P11 that isclassified as an object of a structure is displayed on the upper left. Abuilding image P21 that is classified as an object of a structure, amountain image P22 that is covered with snow on the top and isclassified as an object of a mountain, a mountain image P23 that has twopeaks and is classified as an object of a structure, a passenger carimage P24 that is classified as an object of an automobile, a bus imageP25 that is classified as an object of an automobile is displayed, and aface image P26 that is classified as an object of a person are displayedin this order to the right of the image P21.

In FIG. 9, when the specified position is a position indicated by theposition P, the images P11 and P21 to P26, which are classified as anobject at the position, are displayed in the form of a horizontal bar(bar-shaped display field). When the image at the upper left corner isselected by moving it in the horizontal direction, the time-seriesimages recognized as having the same object are displayed in such a wayas to display the images P11 to P16. As a result, when the user capturesan image with the glasses-mounted camera 11 while walking the sameroute, it is possible to observe a change of the same object like as itis captured with a fixed-point camera. The selection of these imagesmakes it possible to generate a command to start the reproduction fromthe reproduction position of the moving image file including theselected frame image. This process allows the user to select an objectthat the user wants to view among the moving image files and to specifythe desired time. Thus, the user can instruct a reproduction process byspecifying the position to start the reproduction adapted for thepurpose from a large amount of moving image files.

In step S93, the reproduction management unit 94 controls thecommunication unit 91 to transmit the reproduction image selection imagecalled My Map as shown in FIG. 9 to the reproduction terminal 12 via thenetwork 13.

In step S81, if the specified position information is not transmitted,steps S82 to S93 are skipped.

In step S94, the reproduction management unit 94 determines whether aninstruction to terminate the operation by an operation unit (not shown)is issued. If it is determined that an instruction to terminate is notissued, the process returns to step S81 and the subsequent steps arerepeated. In step S94, if it is determined that an instruction toterminate the operation is issued, the process ends.

On the other hand, in the reproduction terminal 12, in step S65, thecontrol unit 71 controls the communication unit 74 to obtain displayimage information called My Map that is the reproduction image selectionimage transmitted from the server 14.

In step S66, the control unit 71 displays a display image called My Map,which is the reproduction image selection image received through thecommunication unit 74, on the display unit 72.

In step S67, the control unit 71 determines whether an instruction toterminate the operation is issued by the operation unit 73. If it isdetermined that an instruction to terminate the operation is not issued,the process returns to step S62 and the subsequent steps are repeated.On the other hand, in step S67, if it is determined that an instructionto terminate the operation is issued, the process ends.

In step S63, if it is determined that the specified position is not aposition specified initially and is not changed, the process of stepsS64 to S66 is repeated. In other words, when the display image selectionimage is displayed on the display unit 72, a state where the My Map thatis the same display image selection image is displayed is kept, as longas the specified position is not changed.

This process allows the moving image file to be reproduced by specifyinga position on the route displayed according to the image-capturedposition when a large amount of images are captured with theglasses-mounted camera 11. Thus, the user can appropriately select thereproduction position of a moving image without memorizing the captureddate and time or the like. The XrossMediaBar displayed by the images P11to P16 and P21 to P26 makes it possible to specify a reproduction startposition of the moving image file from a desired object orimage-captured time to reproduce it.

<Route Correction Process>

A route correction process will be described with reference to theflowchart of FIG. 11. The route correction process is based on theassumption that the My Map that is the display image selection image isdisplayed.

In step S101, the control unit 71 determines whether the correction knobM is moved and the correction scale is operated by a control of theoperation unit 73. If it is determined that the correction knob M ismoved and the correction scale is operated in step S101, the processproceeds to step S102.

In step S102, the control unit 71 controls the communication unit 74 totransmit information indicating a position on the arrow-shapedcorrection scale shown in FIG. 9 of the operated correction knob M tothe server 14 via the network 13.

In step S121, the route editing unit 115 of the reproduction managementunit 94 determines whether operation position information of thecorrection knob M that is the detail of the operation of the correctionscale is transmitted by controlling the communication unit 91. In stepS121, for example, if it is determined that information indicating theposition of the correction knob M is transmitted by the process in stepS102, the process proceeds to step S122.

In step S122, the converted moving image file generation unit 116 readsa moving image file in which a position plotted on the route L displayedin the display image selection image transmitted to the reproductionterminal 12 is included in the index of the image-captured position fromthe storage unit 93. The converted moving image file generation unit 116generates a new converted image to be changed to the predetermined typesof angles by performing an angle of view conversion or rotationconversion on the frame image included in the read moving image file byusing the preceding and succeeding images or the like. Morespecifically, the converted moving image file generation unit 116generates the converted image by performing an angle of view shift,homography transformation, viewpoint composition, or the like.

In other words, the converted moving image file generation unit 116converts images of frame images Org(N−1) to Org(N+1) included in anoriginal moving image file stored in the storage unit 93 as shown in theupper part of FIG. 12 into images of converted images P1(N−1) to P1(N+1)as shown by Pattern1 in the middle part and converted images P2(N−1) toP2(N+1) as shown by Pattern2 in the lower part of FIG. 12, by performingan angle of view conversion or rotation conversion using the surroundingimages.

The converted moving image file generation unit 116, when generating aconverted image, may generate a converted image with high resolution byusing pixels generated by the phase shift equal to or smaller than apixel in the preceding and succeeding images of a target frame image tobe processed.

In step S123, the node difference calculation unit 117 calculates a nodedifference by obtaining a difference between each frame image generatedusing an angle of view conversion and rotation conversion and anoriginal frame image. In other words, the node difference calculationunit 117 calculates a node difference Error_n(Org(N),P1(N)) between theoriginal image Org(N) and the converted image P1(N) as shown in FIG. 12.The difference to be calculated may be a sum total of pixel valuedifferences between pixels at the same position of each image. Thedifference to be calculated may also be a sum total of pixel valuedifferences between pixels at the same position of each image of animage Org(N)′ obtained by compensating the image Org(N) by the globalmotion compensation technique and the converted image P1. The differenceto be calculated may also be an average value or variance of opticalflows of the original image Org(N) and the converted image P1(N).

In step S124, the edge difference calculation unit 118 calculates adifference between the preceding and succeeding images for each of allpatterns when the converted images, which are generated by the convertedmoving image file generating unit 116 and are obtained by performingvarious angle of view conversion and rotation conversion processing, areplaced in time series in various ways. In other words, as shown in FIG.13, the image Org(N), the converted image P1(N), and the converted imageP2(N) are arranged as images placed next to the original image Org(N−1),and thus a difference between the preceding and succeeding patterns forall the patterns is calculated as an edge difference. In other words, inthis case, Error_e(Org(N−1),Org(N)), Error_e(Org(N−1),P1(N)), andError_e(Org(N−1),P2(N)) are calculated. For the difference obtained inthis process, a technique similar to the calculation of difference usedto calculate the node difference may be used, or other techniques may beused.

In step S125, the evaluation value calculation unit 119 calculates anevaluation value E using a sum total of node differences Error_nAll anda sum total of edge differences Error_eAll of the frame image arrangedfor each pattern of all the patterns. The sum total of node differencesError_nAll and the sum total of edge differences Error_eAll arerespectively calculated using a node difference and an edge differenceobtained by combining each arrangement pattern of the image Org, theimage P1, and the image P2. The evaluation value E is represented by thefollowing Equation.

E=(1−W)×Error_eAll+W×Error_nAll

In the above Equation, E is an evaluation value for the entire movingimage file when the frame image and the converted image are arranged intime series in a predetermined pattern, w is a weight corresponding tothe amount of movement of the correction knob M and ranges from 0 to 1,Error_nAll is the sum total of node differences, and Error_eAll is thesum total of edge differences.

In other words, the evaluation value E, which is set according to thearrangement of frame image in the moving image file generated in variouspatterns, may be processed as a graph optimization problem obtained fromthe node difference, that is, the difference with the original image foreach image that may be an option of the frame images that constitute themoving image file and the edge difference between the preceding andsucceeding images that may be an option that can be arranged in timeseries, as shown in FIG. 14. In FIG. 14, circle (N−1), circle (N), andcircle (N+1) each indicates a node, and arrows indicate an edge.Error_n(Org(N),Org(N)) indicates a node difference between the originalframe image Org(N) and the frame image Org(N). Error_n(Org(N),P1(N))indicates a node difference between the original frame image Org(N) andthe converted image P1(N). Error_n(Org(N),P2(N)) indicates a nodedifference between the original frame image Org(N) and the convertedimage P2(N). Error(Org(N),Org(N−1)), Error(P1(N),P2(N−1)),Errore(P2(N),P2(N−1)), and values among them indicate an edgedifference.

Thus, in step S126, the converted moving image file selection unit 120selects a converted moving image file composed of patterns arranged intime series of the frame image having the lowest evaluation value E,that is, the smallest evaluation value or the converted image. Theconverted moving image file selection unit 120 allows the storage unit93 to store the converted moving image file composed of frame imagesarranged in time series or the converted moving image file composed ofconverted images, according to the selected pattern.

In step S127, the imaging direction coloring unit 112 extracts imagingdirection information of the frame image having the smallest evaluationvalue E or each frame image that constitute the converted moving imagefile composed of patterns arranged in time series of the convertedimage, selected by the converted moving image file selection unit 120.The imaging direction coloring unit 112 determines color arrangement ofthe route L shown in FIG. 9 and generates route display information ofthe route L.

In step S128, the reproduction management unit 94 controls thecommunication unit 91 to transmit the newly generated route displayinformation to the reproduction terminal 12 via the network 13.

In step S129, the reproduction management unit 94 determines whetherinformation instructing to terminate the operation is received throughthe communication unit 91. If it is determined that an instruction toterminate the operation is not issued, the process returns to step S121and the subsequent steps are repeated. In step S121, if it is determinedthat the correction scale information is not transmitted, the process insteps S122 to S128 is skipped. In step S129, if it is determined that aninstruction to terminate the operation is issued, the process ends.

On the other hand, in the reproduction terminal 12, in step S103, thecontrol unit 71 controls the communication unit 74 to obtain the routedisplay information transmitted.

In step S104, the control unit 71 controls the display unit 72 toprocess coloring on the route L shown in FIG. 9 and to display theprocessed route as a new route L on the display unit 72 based on thereceived route display information.

In step S105, the control unit 71 determines whether the termination isinstructed by the operation of the operation unit 73. If it isdetermined that the termination is not instructed, the process returnsto step S101 and the subsequent steps are repeated. Then, in step S105,if it is determined that the termination is instructed, the processends.

This process allows the frame image that constitutes the moving imagefile to be converted into a converted moving image file with a patternhaving the smallest evaluation value E among the combined convertedimages constituted by performing a plurality types of angle of viewconversion and rotation conversion by operating the correction knob M.Such a process makes it possible to recognize a difference between themoving image file composed of an image that is similar to the originalimage and is difficult to be collapsed but that is difficult to view dueto frequent change in imaging directions for each frame image, so-calledmoving image file that is easy to induce motion sickness, and the movingimage file composed of an image that is significantly different from theoriginal image and is easy to be collapsed but that is relatively easyto view due to infrequent change in imaging directions for each frameimage, so-called moving image file that is difficult to induce motionsickness, from a change in color assigned on the route L in response tothe operation of the correction knob M.

When the image collapse occurs in the converted image as describedabove, a difference between the preceding and succeeding images in timeseries is expected to be larger. Thus, for the frame image having thelowest evaluation value E, the frame image that constitutes the movingimage file selected based on the arrangement of the time series of theconverted images, or the frame image that exists in a section in whichthe edge difference is greater than a predetermined threshold Th and asection indicated by margin before and after the section describedabove, for example as shown in FIG. 15, using the edge differenceinformation of the converted image, it may be possible to prevent acollapsed section from being viewed by setting the frame images in asection in which the reproduction is not performed, that is, a cutsection. This allows a collapsed image to be prevented from being viewedwhile selecting a moving image file by color on the route L regarded asoptimal by the user, and thus a converted moving image file that iseasier to view can be generated.

<Reproduction Process>

The reproduction process will be described with reference to theflowchart of FIG. 16. The reproduction process is based on theassumption that the display image selection image as shown in FIG. 9 isdisplayed. The moving image file that is instructed to be reproduced isthe converted moving image file generated by minimizing the evaluationvalue E by the above-described route correction process.

In step S151, the control unit 71 determines whether the moving imagefile is instructed to be reproduced by a predetermined operation bycontrolling the operation unit 73. A similar process is repeated untilthe reproduction is instructed.

In step S151, for example, if the position P on the route L in the MyMap shown in FIG. 9 is selected by a pointer (not shown) and it isdetermined that the reproduction is instructed, the process proceeds tostep S152.

In step S152, the control unit 71 controls the communication unit 74 totransmit position information on the map corresponding to the selectedposition P and a command for instructing to reproduce the moving imagefile located in the position to the server 14.

In step S171, the reproduction management unit 94 of the server 14controls the communication unit 91 to determine whether the reproductionis instructed, and repeats a similar process until the reproduction isinstructed. In step S171, for example, if it is determined thatinformation for instructing to reproduce is transmitted by the processin step S152, it is assumed that the reproduction is instructed, andthen the process proceeds to step S172.

In step S172, the reproduction management unit 94 searches anunprocessed frame image among information of the corresponding movingimage files (converted moving image files) based on the positioninformation and time information in which the reproduction isinstructed, and reads the unprocessed frame image as an image to beprocessed.

In step S173, the prohibition editing unit 121 extracts the read imageto be processed and determines whether a prohibited region exists in theimage from the assigned image-captured time, imaging position, imagingdirection, or the like. In other words, more specifically, theprohibition editing unit 121 determines whether, for example, as aprohibited region, a region containing information such as personallyidentifiable information, a region in which military secret or the likespecified based on position information is seemed to be captured, or aregion that may have a problem if disclosure is made at a predeterminedtime exists in the image captured by the glasses-mounted camera 11.

If it is determined that there is a prohibited region in step S173, theprohibition editing unit 121, in step S174, makes a prohibited region ofan image to be processed incapable of viewing by performing a mosaicprocess or blackening process in the image. If it is determined thatthere is not a prohibited region in step S173, the process in step S174is skipped.

In step S175, the reproduction management unit 94 stores the read imageto be processed or the target image to be processed that is edited asprohibited by the prohibition editing unit 121 in the moving image fileas a reproduction image.

In step S176, the reproduction management unit 94 determines whether anunprocessed frame image exists in the moving image file. If it isdetermined that there is an unprocessed frame image, the process returnsto step S172 and the subsequent steps are repeated. If all the frameimages included in the moving image file corresponding to the time andposition at which the reproduction is instructed are target images to beprocessed that are used to determine whether there is a prohibitedregion, the process proceeds to step S177.

In step S177, the reproduction management unit 94 controls thecommunication unit 91 to transmit the moving image file to which thereproduction is instructed to the reproduction terminal 12, and theprocess returns to step S171.

On the other hand, in the reproduction terminal 12, in step S153, thecontrol unit 71 controls the communication unit 74 to receive the movingimage file transmitted from the server 14 via the network 13.

In step S154, the control unit 71 sequentially reproduces the movingimage files received through the communication unit 74 in frame imageunits and displays the moving image files on the display unit 72.

In the above, although an example in which the reproduction isinstructed by selecting the position P on the route L shown on the MyMap of FIG. 9 using the operation unit 73 is described, the reproductionmay be similarly instructed by selecting any of the images P11 to P16and P21 to P26 in the XrossMediaBar.

The process described above allows the user to specify a reproductionposition on the moving image file that is captured with theglasses-mounted camera 11 and is displayed as the route L in the form ofdisplay image selection image as shown in FIG. 9, called My Map,according to the position and object on the map or the time at which theobject is captured. The route L shown in FIG. 9 is displayed as an imageobtained by adding different colors to each imaging direction of theframe image included in the moving image file captured on the route, andthus a change in the imaging directions, that is, a degree of blurbetween frame images in the moving image file can be visually recognizedat a glance. The operation of the correction knob M allows aninstruction of reproduction while viewing color assigned on the route Lby recognizing a difference between the moving image file composed of animage that is similar to the original image and is difficult to becollapsed but that is difficult to view due to frequent change inimaging directions for each frame image, so-called moving image filethat is easy to induce motion sickness, and the moving image filecomposed of an image that is significantly different from the originalimage and is easy to be collapsed but that is relatively easy to viewdue to infrequent change in imaging directions for each frame image,so-called moving image file that is difficult to induce motion sickness.

In the above, although an example in which the glasses-mounted camera 11is used in the image capturing process is described, an image with lessblur can be generated by performing a similar process on an imagecaptured with a video camera or other imaging devices as long as theimaging devices are equipped with a function of capturing a movingimage. In the above, although an example in which the server 14 on thenetwork 13 performs various types of processing is described, thefunction of the server 14 may be implemented by a cloud computing orother approaches as long as they are equipped with the same function asthe server 14.

Incidentally, the above series of processes can, for example, beexecuted by hardware, or can be executed by software. In the case wherethe series of processes is executed by software, a program configuringthis software is installed in a computer included in dedicated hardware,or a general-purpose personal computer which can execute variousfunctions when various programs are installed, etc., from a recordingmedium.

FIG. 17 shows an example configuration of a general-purpose personalcomputer. The computer includes a central processing unit (CPU) 1001. Aninput/output interface 1005 is connected to the CPU 1001 through a bus1004. A read only memory (ROM) 1002 and a random access memory (RAM)1003 are connected to the bus 1004.

An input unit 1006 including an input device, such as a keyboard, amouse, etc., which is used by the user to input an operation command, anoutput unit 1007 which outputs a process operation screen or an image ofa process result to a display device, a storage unit 1008 including ahard disk drive etc. which stores a program or various items of data,and a communication unit 1009 including a local area network (LAN)adaptor etc. which performs a communication process through a networktypified by the Internet, are connected to the input/output interface1005. Also, connected is a drive 1010 which reads and writes data fromand to a removable medium 1011, such as a magnetic disk (including aflexible disk), an optical disk (including a compact disc-read onlymemory (CD-ROM) and a digital versatile disc (DVD)), an magneto-opticaldisk (including an mini disc (MD)), or a semiconductor memory, etc.

The CPU 1001 executes various processes according to a program stored inthe ROM 1002 or a program which is read from the removable medium 1011,such as a magnetic disk, an optical disk, a magneto-optical disk, or asemiconductor memory, etc., is installed in the storage unit 1008, andis loaded from the storage unit 1008 to the RAM 1003. The RAM 1003 alsostores data which is required when the CPU 1001 executes variousprocesses, etc., as appropriate.

In the computer configured as described above, the CPU 1001 loads aprogram that is stored, for example, in the storage unit 1008 onto theRAM 1003 via the input/output interface 1005 and the bus 1004, andexecutes the program. Thus, the above-described series of processing isperformed.

Programs to be executed by the computer (the CPU 1001) are providedbeing recorded in the removable medium 1011 which is a packaged mediumor the like. Also, programs may be provided via a wired or wirelesstransmission medium, such as a local area network, the Internet ordigital satellite broadcasting.

In the computer, by inserting the removable medium 1011 into the drive1010, the program can be installed in the storage unit 1008 via theinput/output interface 1005. Further, the program can be received by thecommunication unit 1009 via a wired or wireless transmission medium andinstalled in the storage unit 1008. Moreover, the program can beinstalled in advance in the ROM 1002 or the storage unit 1008.

It should be noted that the program executed by a computer might be aprogram that is processed in time series according to the sequencedescribed in this specification or a program that is processed inparallel or at necessary timing such as upon calling.

Further, in the present disclosure, a system has the meaning of a set ofa plurality of configured elements (such as an apparatus or a module(part)), and does not take into account whether or not all theconfigured elements are in the same casing. Therefore, the system may beeither a plurality of apparatuses, stored in separate casings andconnected through a network, or a plurality of modules within a singlecasing.

An embodiment of the present technology is not limited to theembodiments described above, and various changes and modifications maybe made without departing from the scope of the present technology.

For example, the present technology can adopt a configuration of cloudcomputing which processes by allocating and connecting one function by aplurality of apparatuses through a network.

Further, each step described by the above-mentioned flowcharts can beexecuted by one apparatus or by allocating a plurality of apparatuses.

In addition, in the case where a plurality of processes are included inone step, the plurality of processes included in this one step can beexecuted by one apparatus or by sharing a plurality of apparatuses.

Additionally, the present technology may also be configured as below.

(1)

An image processing device including:

an acquisition unit configured to acquire information on an imagingposition and an imaging direction in units of frame images thatconstitute a moving image obtained through capturing by an imaging unit;

a converted image generation unit configured to generate a plurality ofconverted images having different imaging directions for each frameimage that constitutes the moving image based on the frame image itselfand preceding and succeeding frame images of the frame image;

an evaluation value calculation unit configured to calculate anevaluation value for each converted moving image constituted bycombining the converted image and the original frame image, theevaluation value being used to evaluate a blur between the convertedimages or between the original frame images; and a selection unitconfigured to select a converted moving image with less blur based on anevaluation value calculated by the evaluation value calculation unit.

(2)

The image processing device according to (1), further including:

a node difference calculation unit configured to calculate a differencebetween the converted image generated by the converted image generationunit and the original frame image as a node difference; and

an edge difference calculation unit configured to calculate a differencein a time series including the converted image generated by theconverted image generation unit and the original frame image as an edgedifference,

wherein the evaluation value calculation unit calculates an evaluationvalue used to evaluate a blur for each of converted moving images basedon each of a sum of the edge difference and a sum of the node differencebetween the converted images or the original frame images included inthe converted moving image constituted by combining the converted imageand the original image.

(3)

The image processing device according to (2),

wherein the evaluation value calculation unit calculates an evaluationvalue used to evaluate a blur for each of the converted moving images byapplying a weight to each of the sum of the edge difference and the sumof the node difference between the converted images or the originalframe images included in the converted moving image constituted bycombination of the converted image and the original image and bycalculating a sum of the weighted values.

(4)

The image processing device according to (3), further including:

a weight input unit configured to input a weight to be applied to thesum of the edge difference and the sum of the node difference, a sum ofthe weight applied to each of the sum of the edge difference and the sumof the node difference being a value of 1,

wherein the evaluation value calculation unit calculates an evaluationvalue used to evaluate a blur for each of the converted moving images byapplying a weight inputted by the weight input unit to each of the sumof the edge difference and the sum of the node difference between theconverted images or the original frame images included in the convertedmoving image constituted by combination of the converted image and theoriginal image and by calculating a sum of the weighted values.

(5)

The image processing device according to (1),

wherein the storage unit also stores information on an imaging directionbeing changed in accordance with the original frame image by having aplurality of predetermined angles of view and by rotating to a pluralityof angles in association with the converted image when the convertedimage is generated by the converted image generation unit, and

wherein the image processing device further includes a display imagegeneration unit configured to generate a display image that displays aroute on a map based on the converted moving image or the imagingposition information of the frame image of the moving image and displaysby applying a different color to each imaging direction on the routebased on the imaging direction information.

(6)

The image processing device according to (5), further including:

a specifying unit configured to specify a position on the routedisplayed on the map on the display image; and

a reproduction unit configured to reproduce the converted moving imagethat constitutes the route or the moving image from a frame imageincluding information on an imaging position corresponding to a positionspecified by the specifying unit.

(7)

The image processing device according to (1),

wherein the converted image generation unit generates a converted imageby converting each frame image that constitutes a moving image stored inthe storage unit into a plurality of images having the different imagingdirections by having a plurality of predetermined angles of view and byrotating to a plurality of angles by a technique including an angle ofview shift, homography transformation, and viewpoint composition, usingthe frame image itself and preceding and succeeding frame images of theframe image.

(8)

The image processing device according to (1),

wherein the converted image generation unit generates a plurality ofhigher-resolution converted images having the different imagingdirections for each frame image that constitutes a moving image storedin the storage unit by using a pixel generated by a phase shift equal toor smaller than a pixel based on the frame image itself and precedingand succeeding frame images of the frame image.

(9)

The image processing device according to any one of (1) to (8), furtherincluding:

the imaging unit configured to capture the moving image,

wherein the acquisition unit stores the information on the imagingposition and the imaging direction of the frame image in associationwith each of the frame images.

(10)

An image processing method including:

acquiring information on an imaging position and an imaging direction inunits of frame images that constitute a moving image obtained throughcapturing by an imaging unit;

generating a plurality of converted images having different imagingdirections for each frame image that constitutes the moving image basedon the frame image itself and preceding and succeeding frame images ofthe frame image;

calculating an evaluation value for each converted moving imageconstituted by combining the converted image and the original frameimage, the evaluation value being used to evaluate a blur between theconverted images or between the original frame images; and

selecting a converted moving image with less blur based on a calculatedevaluation value.

(11)

A program for causing a computer to execute processing including:

an acquisition step of acquiring information on an imaging position andan imaging direction in units of frame images that constitute a movingimage obtained through capturing by an imaging unit;

a converted image generation step of generating a plurality of convertedimages having different imaging directions for each frame image thatconstitutes the moving image based on the frame image itself andpreceding and succeeding frame images of the frame image;

an evaluation value calculation step of calculating an evaluation valuefor each converted moving image constituted by combining the convertedimage and the original frame image, the evaluation value being used toevaluate a blur between the converted images or between the originalframe images; and

a selection step of selecting a converted moving image with less blurbased on an evaluation value calculated by processing of the evaluationvalue calculation step.

REFERENCE SIGNS LIST

-   11, 11-1 to 11-N glasses-mounted camera-   12, 12-1 to 12-M reproduction terminal-   13 network-   14 server-   32 imaging unit-   33 control unit-   51 recording unit-   52 GPS-   53 RTC-   54 earth's axis sensor-   55 gyro sensor-   56 controller-   57 communication unit-   71 control unit-   72 display unit-   73 operation unit-   74 communication unit-   75 display image storage unit-   91 communication unit-   92 recording unit-   93 storage unit-   94 reproduction management unit-   101 indexing unit-   111 My Map creation unit-   112 imaging direction coloring unit-   113 object recognition unit-   114 map data storage unit-   115 route editing unit-   116 converted moving image file generation unit-   117 node difference calculation unit-   118 edge difference calculation unit-   119 evaluation value calculation unit-   120 converted moving image file selection unit-   121 prohibition editing unit

1. An image processing device comprising: at least one processorconfigured to: acquire information on a first frame image, a secondframe image and a third frame image that constitute a moving imageobtained through capturing by an imaging unit, the second frame imagebeing between the first frame image and the third frame image; correct asecond imaging direction of the second frame image based on at least oneof a first imaging direction of the first frame image and a thirdimaging direction of the third frame image to reduce differences betweenthe first imaging direction, the second imaging direction and the thirdimaging direction; and send the second frame image having the correctedsecond imaging direction to a display unit.
 2. The image processingdevice according to claim 1, wherein the at least one processor isfurther configured to: acquire a correction input by a user to determinea correction amount of the second imaging direction; and correct thesecond imaging direction based on the correction amount.
 3. The imageprocessing device according to claim 2, wherein the at least oneprocessor is further configured to control the display unit to display acorrection indicator indicating the correction amount.
 4. The imageprocessing device according to claim 3, wherein the at least oneprocessor is further configured to control the display unit to display aroute corresponding to the correction amount along with the correctionindicator.
 5. The image processing device according to claim 3, thecorrection indicator is an operable scale receiving the correctioninput.
 6. The image processing device according to claim 1, wherein theat least one processor is further configured to control the display unitto display a route on a map, and wherein the route corresponds to frameimages that constitute the moving image.
 7. The image processing deviceaccording to claim 6, wherein the route has colors corresponding toimaging directions of the frame images, and wherein changes between thecolors correspond to changes between the imaging directions.
 8. Theimage processing device according to claim 6, wherein the at least oneprocessor is further configured to: recognize at least one objectincluded in the frame images; and control the display unit to displaythe at least one object on the map as time series images.
 9. The imageprocessing device according to claim 8, wherein the at least one objectincludes a plurality of objects classified into object types includingat least two of a building, a landscape, a vehicle and a person, andwherein the at least one processor is further configured to control thedisplay unit to display the time series images in a first direction anddisplay the plurality of objects classified into the object types in asecond direction perpendicular to the first direction.
 10. The imageprocessing device according to claim 1, wherein the image processingdevice is a network server further comprising a storage unit holding themoving image obtained through capturing by the imaging unit.
 11. Theimage processing device according to claim 10, wherein the at least oneprocessor is further configured to: determine whether frame imagesconstituting the moving image contain a prohibited region to beprohibited from being reproduced; and make the prohibited region to beprocessed incapable of viewing by a user in a case when determining thatthe prohibited region is contained in the frame images.
 12. The imageprocessing device according to claim 1, wherein the at least oneprocessor is further configured to: correct the second imaging directionby generating a converted frame image based on an angle of view shiftusing the second frame image and the first frame image and the thirdframe image.
 13. The image processing device according to claim 1,wherein the at least one processor is further configured to: correct thesecond imaging direction by generating a converted frame image based ontransformation of a shape of the second frame image using the secondframe image and the first frame image and the third frame image.
 14. Theimage processing device according to claim 1, wherein the at least oneprocessor is further configured to: correct the second imaging directionby generating a converted frame image based on composition of viewpointsof the first frame image, the second frame image and the third frameimage.
 15. The image processing device according to claim 1, wherein theimaging unit is a mobile camera.
 16. The image processing deviceaccording to claim 15, wherein the mobile camera is a wearable camera.17. The image processing device according to claim 1, wherein the atleast one processor constitute at least one network server.
 18. Theimage processing device according to claim 1, wherein the at least oneprocessor performs the correcting of the second imaging direction of thesecond frame image and the sending of the second frame image by a cloudcomputing.
 19. An image processing method, performed via at least oneprocessor, the method comprising: acquiring information on a first frameimage, a second frame image and a third frame image that constitute amoving image obtained through capturing by an imaging unit, the secondframe image being an frame image between the first frame image and thethird frame image; correcting a second imaging direction of the secondframe image based on a first imaging direction of the first frame imageand a third imaging direction of the third frame image to reducedifferences between the first imaging direction, the second imagingdirection and the third imaging direction; and sending the second frameimage having the corrected second imaging direction to a display unit.20. A non-transitory computer-readable medium having embodied thereon aprogram, which when executed by a computer, causes the computer toexecute a method, the method comprising: acquiring information on afirst frame image, a second frame image and a third frame image thatconstitute a moving image obtained through capturing by an imaging unit,the second frame image being an frame image between the first frameimage and the third frame image; correcting a second imaging directionof the second frame image based on a first imaging direction of thefirst frame image and a third imaging direction of the third frame imageto reduce differences between the first imaging direction, the secondimaging direction and the third imaging direction; and sending thesecond frame image having the corrected second imaging direction to adisplay unit.